Graduate Research Assistantship(Sagar Mehta) Twitter Text Analysis
Business Problem:¶
Business Problem:
- Domestic abuse, also called "domestic violence" or "intimate partner violence", can be defined as a pattern of behavior in any relationship that is used to gain or maintain power and control over an intimate partner.
- Abuse is physical, sexual, emotional, economic or psychological actions or threats of actions that influence another person.
- This includes any behaviors that frighten, intimidate, terrorize, manipulate, hurt, humiliate, blame, injure, or wound someone.
- Domestic abuse can happen to anyone of any race, age, sexual orientation,religion, or gender.
- It can occur within a range of relationships including couples who are married, living together or dating.
- Domestic violence affects people of all socioeconomic backgrounds and education levels.
- Our goal is to identify different factors that cause domestic violence, especially Substance Abuse and Mental Health.
Work Accomplished¶
Work Done:
- During 4 months we have tried to establish relation between Domestic Violence and other factors cuasing it
- Scraping Data from Twitter using keywords like Domestic Violence and Substance Abuse
- Scraping done with the help of snscrape and twint, click on the link for more information
- Data preprocessing:
- Tokenizing
- Removing stopwords,punctuation,numbers
- Converting to lower case
- Lemmatizing
- Stemming
- Tokenizing
- Exploratory Data Analysis:
- Word Cloud
- N-gram analysis
- Top keywords frequency count
- Topic Modeling
- Time Series Analysis
- Researching Domestic Violence Organizations
Week 1: Domestic Violence¶
Data Introduction and Steps followed¶
Step 1: Read the libraries¶
In this step we import the required fundamental libraries
In [67]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
We have scraped data for the keyword Domestic Violence.
- The total number of 100,000 tweets scraped
- The library used is snscrape
In [5]:
dv = pd.read_csv('domestic violence_week1.csv')
dv.sample(5)
Out[5]:
The dataset contains the following information:
| Field | Description |
|---|---|
| Datetime | The date and time information when the tweet was tweeted |
| Tweet Id | The unique tweet identification number |
| Text | The actual tweet |
| Username | The username who actually tweeted |
| Like Count | The count of likes a tweet got |
| Display Name | The actual name of the user |
| Language | The language in which the tweet was made |
Data Preprocessing¶
The following are the steps we followed to clean the data:
- Create a function that does the following task
- Tokenization
- Remove Special Characters
- Remove numbers
- Convert to lowercase
- Tokenization
These steps are very important when it comes to cleaning text data
In [7]:
def tweet_cleaner(text):
from nltk.tokenize import WordPunctTokenizer
tokenizer = WordPunctTokenizer()
import re
text=re.sub(r'@+','',str(text))
text=re.sub('http?://[A-Za-z0-9./]+','',str(text))
text=re.sub("[^a-zA-Z]", " ",str(text))
lower_case = text.lower()
words = tokenizer.tokenize(lower_case)
return (" ".join(words)).strip()
We need to further clean the data and we use nltk
We follow the following steps:
- Import nltk library and import the required functionality
- Tokenize
- Remove stopwords
- Lemmatize
In [52]:
import nltk
from nltk.corpus import stopwords
from string import punctuation
from nltk.stem import wordnet
from nltk.stem import WordNetLemmatizer
#Tokenize Function
def Tokenize(string):
tokens=nltk.tokenize.word_tokenize(string)
return " ".join(tokens)
#RemoveStopWordsFunction
def RemoveStopWords(string):
#Removing Punctuations
for each in punctuation:
string=string.replace(each,"")
#Removing Stopwords
english_stopwords=stopwords.words('english')
stopwords_removed_tokens=[]
words=string.split(" ")
for each in words:
if each not in english_stopwords:
stopwords_removed_tokens.append(each)
return " ".join(stopwords_removed_tokens)
#LemmatizeFunction
def Lemmatize(string):
word_lem=WordNetLemmatizer()
words=string.split()
lemmatizeWords=[]
for each in words:
lemmatizeWords.append(word_lem.lemmatize(each))
return " ".join(lemmatizeWords)
def Refine(string):
return Lemmatize(RemoveStopWords(Tokenize(string)))
In [9]:
tweets = dv.loc[:,['Datetime','Text']] # we make a new dataframe with only required columns
In [10]:
tweets['text_cleaned'] = tweets['Text'].apply(tweet_cleaner)
tweets.head()
Out[10]:
In [11]:
tweets['clean-text_removed'] = tweets['text_cleaned'].apply(Refine)
tweets.head()
Out[11]:
Exploratory Data analysis¶
Word cloud¶
In [14]:
from PIL import Image
import os
from os import path,getcwd
image_mask = np.array(Image.open('USA MAP BLACK.png'))
d = os.getcwd()
from wordcloud import WordCloud, STOPWORDS,ImageColorGenerator
text = tweets['clean-text_removed'].values
plt.figure(figsize=(15,15))
stopwords = set(STOPWORDS)
image_wc = WordCloud(background_color='black',max_words=250000,stopwords=stopwords,mask=image_mask,colormap='twilight_shifted')
image_wc.generate(str(text))
plt.imshow(image_wc,interpolation='bilinear')
plt.axis("off")
plt.show()
Keyword Analysis¶
The goal was to understand presence of certain words and then define any correlation between these words and domestic violence.
In [15]:
keywords = ['Alcoholism','complications','Substance-related','disorders','complications',
'Family relations','Spouse abuse','Substance abuse','Substance use', 'Intimate Partner Violence',
'cycles of escalation','Duluth model', 'batterer intervention programs',
'intimate partner violence','perpetrator','recidivism','intimidation','threats','physical violence',
'sexual violence','isolation','economic abuse','stalking','psychological abuse',
'coercion related to mental health']
In [16]:
import re
pattern = '|'.join(f"\\b{k}\\b" for k in keywords) # Whole words only
matches = {k: 0 for k in keywords}
for title in tweets['Text']:
for match in re.findall(pattern, title):
matches[match] += 1
print(matches)
In [17]:
match = pd.DataFrame(matches,index=[0])
match.T.sort_values(0).plot(kind = 'barh',legend = False,
figsize = (15,8),title = 'Keywords analysis')
plt.show()
From the keyword analysis above we come to the following conclusion:
- Stalking, perpetrator, sexual violence, threats, intimate partner violence are some of the most used keywords that occur in Domestic violence based tweets
- While Spouse abuse, Family relations and Subtance related the least
- From this keyword analysis we can say that substance abuse and domestic violence have a weak correlations
User analysis¶
In [18]:
dv.groupby(['Username','Display Name'])['Text'].count().sort_values(ascending = False)[:10].plot(kind = 'barh')
plt.show()
The Top 10 usernames are mostly organizations that support victims of Domestic violence.
Topic Modeling¶
- In machine learning and natural language processing, a topic model is a type of statistical model for discovering the abstract "topics" that occur in a collection of documents.
- Topic modeling is a frequently used text-mining tool for discovery of hidden semantic structures in a text body.
- Intuitively, given that a document is about a particular topic, one would expect particular words to appear in the document more or less frequently: "dog" and "bone" will appear more often in documents about dogs, "cat" and "meow" will appear in documents about cats, and "the" and "is" will appear approximately equally in both.
- A document typically concerns multiple topics in different proportions; thus, in a document that is 10% about cats and 90% about dogs, there would probably be about 9 times more dog words than cat words.
The "topics" produced by topic modeling techniques are clusters of similar words.
A topic model captures this intuition in a mathematical framework, which allows examining a set of documents and discovering, based on the statistics of the words in each, what the topics might be and what each document's balance of topics is.
Topic models are also referred to as probabilistic topic models, which refers to statistical algorithms for discovering the latent semantic structures of an extensive text body.
- In the age of information, the amount of the written material we encounter each day is simply beyond our processing capacity.
- Topic models can help to organize and offer insights for us to understand large collections of unstructured text bodies.
- Originally developed as a text-mining tool, topic models have been used to detect instructive structures in data such as genetic information, images, and networks.
Source: Wikipedia
We use LDA based Topic modeling.
LDA stands for Latent Dirichlet Allocation.
- In natural language processing, the latent Dirichlet allocation (LDA) is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar.
- For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word's presence is attributable to one of the document's topics. LDA is an example of a topic model.
We import LDA from Sklearn. Please click on the link for documentation
In [19]:
from sklearn.decomposition import LatentDirichletAllocation
lda = LatentDirichletAllocation(n_components=5,random_state=42)
In [20]:
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(max_df=0.95,min_df=2,stop_words='english')
dtm = cv.fit_transform(tweets['clean-text_removed'])
In [21]:
lda.fit(dtm)
Out[21]:
In [22]:
for i,topic in enumerate(lda.components_):
print(f"The top 15 words for the topic #{i}")
print([cv.get_feature_names()[index] for index in topic.argsort()[-15:]])
In [23]:
topic_results = lda.transform(dtm)
In [24]:
tweets['topic_result_lda'] = topic_results.argmax(axis = 1)
tweets.sample(5)
Out[24]:
In [44]:
topic_mapping = {4:'Discussion',1:'News',3:'Campaign and Law',2:'Help and services',0:'Arrests & Court Cases'}
In [45]:
tweets['topic'] = tweets.topic_result_lda.map(topic_mapping)
In [39]:
tweets.sample(5)
Out[39]:
In [46]:
plt.figure(figsize=(15,8))
tweets.topic.value_counts().plot(kind = 'bar')
plt.xlabel('Topics')
plt.ylabel('counts')
plt.title('Different topics')
plt.xticks(rotation = 0)
plt.show()
Conclusion from topic modeling:
- Most of the Domestic violence tweets are Discussion based tweets
- Second most tweets are related to the topic Help and services
- Then there are news related topics
- Then there are Arrest & court Cases based topics and finally Campaign and Law based
Week 2: Substance Abuse¶
In [ ]:
We follow similar steps and do the same procedure and scrape tweets as we did for domestic violence
In [47]:
sa = pd.read_csv('substance abuse_week2.csv',lineterminator='\n')
sa.sample(5)
Out[47]:
In [48]:
tweets = sa.loc[:,['Datetime','Text']]
In [49]:
tweets['text_cleaned'] = tweets['Text'].apply(tweet_cleaner)
tweets.head()
Out[49]:
In [53]:
tweets['clean-text_removed'] = tweets['text_cleaned'].apply(Refine)
tweets.head()
Out[53]:
In [54]:
from PIL import Image
from wordcloud import WordCloud, STOPWORDS,ImageColorGenerator
image_mask = np.array(Image.open('USA MAP BLACK.png'))
text = tweets['clean-text_removed'].values
plt.figure(figsize=(15,15))
stopwords = set(STOPWORDS)
image_wc = WordCloud(background_color='black',max_words=250000,stopwords=stopwords,mask=image_mask,colormap='twilight_shifted')
image_wc.generate(str(text))
plt.imshow(image_wc,interpolation='bilinear')
plt.axis("off")
plt.show()
In [55]:
keywords = ['Alcoholism','complications','Domestic abuse','disorders','complications',
'Family relations','Spouse abuse','domestic violence','Domestic Violence', 'Intimate Partner Violence',
'intimate partner violence','perpetrator','recidivism','intimidation','threats','physical violence',
'sexual violence','isolation','stalking','psychological abuse',
'mental health']
In [56]:
import re
pattern = '|'.join(f"\\b{k}\\b" for k in keywords) # Whole words only
matches = {k: 0 for k in keywords}
for title in tweets['Text']:
for match in re.findall(pattern, title):
matches[match] += 1
print(matches)
In [57]:
match = pd.DataFrame(matches,index=[0])
match.T.sort_values(0).plot(kind = 'barh',legend = False,
figsize = (15,8),title = 'Keywords analysis')
plt.show()
From this figure above we come to the following conclusion:
- Mental health, disorders, domestic violence are the most used keywords along with Substance Abuse
- Family relations, Spouse abuse, and Intimate partner Violence the least.
In [58]:
from sklearn.decomposition import LatentDirichletAllocation
lda = LatentDirichletAllocation(n_components=5,random_state=42)
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(max_df=0.95,min_df=2,stop_words='english')
dtm = cv.fit_transform(tweets['clean-text_removed'])
In [59]:
lda.fit(dtm)
Out[59]:
In [60]:
for i,topic in enumerate(lda.components_):
print(f"The top 15 words for the topic #{i}")
print([cv.get_feature_names()[index] for index in topic.argsort()[-15:]])
In [61]:
topic_results = lda.transform(dtm)
In [62]:
tweets['topic_result_lda'] = topic_results.argmax(axis = 1)
tweets.sample(5)
Out[62]:
In [63]:
topic_mapping = {0:'Discussion',1:'Causes and effects',2:'Treatment and Remedies',3:'Awareness',4:'News'}
In [64]:
tweets['topic'] = tweets.topic_result_lda.map(topic_mapping)
In [65]:
plt.figure(figsize=(15,8))
tweets.topic.value_counts().plot(kind = 'bar')
plt.xlabel('Topics')
plt.ylabel('counts')
plt.title('Different topics')
plt.xticks(rotation = 0)
plt.show()
Conclusion from topic modeling:
- Most of the Substance abuse tweets are Awareness based tweets
- Second most tweets are related to the topic Treatment and Remedies
- Then there are causes and effect related topics
- Then there are News based topics and finally Campaign and Law based
In [68]:
sa.groupby(['Username','Display Name'])['Text'].count().sort_values(ascending = False)[:10].plot(kind = 'barh')
plt.show()
The Top 10 usernames are mostly organizations that support victims of Substance abuse
Week 3
Scrape data for keyword substance abuse using snscrape
The chunks below show dummy code for how to scrape using snscrape
In [2]:
import snscrape.modules.twitter as sntwitter
import pandas
# Creating list to append tweet data to
tweets_list2 = []
# Using TwitterSearchScraper to scrape data and append tweets to list
for i,tweet in enumerate(sntwitter.TwitterSearchScraper('substance abuse since:2021-01-01 until:2021-03-25').get_items()):
if i>50:
break
tweets_list2.append([tweet.date, tweet.id, tweet.content])
# Creating a dataframe from the tweets list above
tweets_dfdummy = pd.DataFrame(tweets_list2, columns=['Datetime', 'Tweet Id', 'Text'])
In [ ]:
In [ ]:
import snscrape.modules.twitter as sntwitter
import pandas
# Creating list to append tweet data to
tweets_list2 = []
# Using TwitterSearchScraper to scrape data and append tweets to list
for i,tweet in enumerate(sntwitter.TwitterSearchScraper('substance abuse since:2021-01-01 until:2021-03-25').get_items()):
if i>50000:
break
tweets_list2.append([tweet.date, tweet.id, tweet.content])
# Creating a dataframe from the tweets list above
tweets_df2 = pd.DataFrame(tweets_list2, columns=['Datetime', 'Tweet Id', 'Text'])
In [ ]:
Scrape data for keyword domestic violence using snsscrape
In [ ]:
import snscrape.modules.twitter as sntwitter
import pandas
# Creating list to append tweet data to
tweets_list7 = []
# Using TwitterSearchScraper to scrape data and append tweets to list
for i,tweet in enumerate(sntwitter.TwitterSearchScraper('domestic violence since:2021-01-01 until:2021-11-21').get_items()):
if i>200000:
break
tweets_list7.append([tweet.date, tweet.id, tweet.content])
# Creating a dataframe from the tweets list above
tweets_df7 = pd.DataFrame(tweets_list7, columns=['Datetime', 'Tweet Id', 'Text'])
In [ ]:
# After merging all the files create a csv file and load to the local computer and reload the files
Read the dataframe and inspect
In [69]:
sa = pd.read_csv('Substance abuse final.csv',lineterminator='\n')
dv = pd.read_csv('Domestic violence final.csv')
In [4]:
sa = pd.read_csv('Substance abuse final.csv',lineterminator='\n')
In [5]:
sa['Datetime'] = pd.to_datetime(sa['Datetime'])
Check for datatypes
In [6]:
sa.dtypes
Out[6]:
Inspect first few columns
In [56]:
sa.head()
Out[56]:
In [57]:
dv.head()
Out[57]:
Select only relevant columns
In [61]:
dv_df = dv.loc[:,['date','Text']]
sa_df = sa.loc[:,['Date','Text']]
In [62]:
sa_df.head()
Out[62]:
In [63]:
dv_df.head()
Out[63]:
Data Preprocessing
In [65]:
dv_df['text_cleaned'] = dv_df['Text'].apply(tweet_cleaner)
sa_df['text_cleaned'] = sa_df['Text'].apply(tweet_cleaner)
In [66]:
dv_df.head(1)
Out[66]:
In [67]:
sa_df.head(1)
Out[67]:
More Data cleaning
In [ ]:
dv_df['clean-text_removed'] = dv_df['text_cleaned'].apply(Refine)
sa_df['clean-text_removed'] = sa_df['text_cleaned'].apply(Refine)
In [69]:
sa_df.head()
Out[69]:
In [70]:
dv_df.head()
Out[70]:
Word cloud
Word cloud for Domestic violence
In [149]:
from wordcloud import WordCloud, STOPWORDS,ImageColorGenerator
image_colors = ImageColorGenerator(image_mask)
from PIL import Image
from wordcloud import WordCloud, STOPWORDS,ImageColorGenerator
image_mask = np.array(Image.open('Publication1-2.jpeg').convert("L"))
text = dv_df['clean-text_removed'].values
wc_dv = WordCloud(stopwords=STOPWORDS,
background_color="white",
mode="RGBA",
max_words=10000,
#contour_width=3,
repeat=True,
mask=image_mask,
colormap='brg')
wc_dv.generate(str(text))
#wc_dv.recolor(color_func = image_colors)
plt.figure(figsize=(15,15))
plt.imshow(wc_dv)
plt.axis("off")
plt.show()
Word cloud for Substance abuse
In [147]:
from PIL import Image
image_mask = np.array(Image.open('sa.jpeg'))
from wordcloud import WordCloud, STOPWORDS,ImageColorGenerator
image_colors = ImageColorGenerator(image_mask)
text = sa_df['clean-text_removed'].values
wc_dv = WordCloud(stopwords=STOPWORDS,
background_color="white",
mode="RGBA",
max_words=100000,
#contour_width=3,
repeat=True,
mask=image_mask,
colormap='gist_yarg')
wc_dv.generate(str(text))
plt.figure(figsize=(15,15))
plt.imshow(wc_dv)
plt.axis("off")
plt.savefig('sv.png')
plt.show()
What are n-grams?
N-grams are continuous sequences of words or symbols or tokens in a document.
In technical terms, they can be defined as the neighbouring sequences of items in a document.
They come into play when we deal with text data in NLP(Natural Language Processing) tasks.
| n | Term |
|---|---|
| 1 | Unigram |
| 2 | Bigram |
| 3 | Trigram |
| n | n-gram |
| Example | Type of ngram |
|---|---|
| ['I','stay' 'in' 'Atlanta'] | Unigram |
| ['I stay','stay in','in Atlanta] | Bigram |
| ['I'stay in'], ['stay in Atlanta] | Trigram |
Source: Analytics vidhya
Ngrams
In [86]:
#dv
plt.style.use('bmh')
import re
import nltk
def basic_clean(text):
wnl = nltk.stem.WordNetLemmatizer()
stopwords = nltk.corpus.stopwords.words('english')
text = text.lower()
words = re.sub(r'[^\w\s]', '', text).split()
return [wnl.lemmatize(word) for word in words if word not in stopwords]
words = basic_clean(''.join(str(dv_df['Text'].tolist())))
plt.figure(figsize=(15,8))
pd.Series(nltk.ngrams(words, 4)).value_counts()[:20].plot(kind = 'barh')
plt.ylabel('N = 4')
plt.title('N-grams(4) analysis')
plt.show()
The words that occur most with domestic violence are awareness, month, sexual,assault
In [87]:
words = basic_clean(''.join(str(sa_df['Text'].tolist())))
plt.figure(figsize=(15,8))
pd.Series(nltk.ngrams(words, 4)).value_counts()[:20].plot(kind = 'barh')
plt.ylabel('N = 4')
plt.title('N-grams(4) analysis')
plt.show()
Since we wanted to understand the relation between substance abuse and domestic violence we just passed those keywords to check for their frequency count
In [89]:
keywords = ['domestic violence','substance abuse','abuse']
import re
pattern = '|'.join(f"\\b{k}\\b" for k in keywords) # Whole words only
matches = {k: 0 for k in keywords}
for title in dv_df['Text']:
for match in re.findall(pattern, str(title)):
matches[match] += 1
print(matches)
In [90]:
keywords = ['domestic violence','substance abuse','violence']
import re
pattern = '|'.join(f"\\b{k}\\b" for k in keywords) # Whole words only
matches = {k: 0 for k in keywords}
for title in sa_df['Text']:
for match in re.findall(pattern, str(title)):
matches[match] += 1
print(matches)
Week 3¶
Time Series¶
What Is a Time Series?
- A time series is a sequence of data points that occur in successive order over some period of time.
This can be contrasted with cross-sectional data, which captures a point-in-time.
In investing, a time series tracks the movement of the chosen data points, such as a security’s price, over a specified period of time with data points recorded at regular intervals.
- There is no minimum or maximum amount of time that must be included, allowing the data to be gathered in a way that provides the information being sought by the investor or analyst examining the activity.
KEY TAKEAWAYS
- A time series is a data set that tracks a sample over time.
- In particular, a time series allows one to see what factors influence certain variables from period to period.
- Time series analysis can be useful to see how a given asset, security, or economic variable changes over time.
- Forecasting methods using time series are used in both fundamental and technical analysis.
- Although cross-sectional data is seen as the opposite of time series, the two are often used together in practice.
Source: Investopedia
In [95]:
dv_df['date'] = pd.to_datetime(dv_df['date'] )
df_dv_ts = dv_df.groupby('date')['Text'].count().reset_index()
plt.style.use('fivethirtyeight')
plt.figure(figsize = (20,10))
sns.lineplot(x = 'date',y = "Text",data =df_dv_ts )
plt.show()
In [97]:
sa_df['Date'] = pd.to_datetime(sa_df['Date'] )
sa_df_ts = sa_df.groupby('Date')['Text'].count().reset_index()
plt.style.use('fivethirtyeight')
plt.figure(figsize = (20,10))
sns.lineplot(x = 'Date',y = "Text",data =sa_df_ts )
plt.show()
In [109]:
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.dates as mdates
import matplotlib.ticker as ticker
import pandas as pd
import seaborn as sns
from datetime import datetime
In [99]:
df2021 =sa_df_ts[sa_df_ts['Date'] > '2020-12-31']
In [125]:
plt.style.use('fivethirtyeight')
plt.figure(figsize=(15,8))
ax1 = sns.lineplot(x = 'Date',y = 'Text',data = df2021,label = 'Substance Abuse Tweets')
ax2 = sns.lineplot(x = 'date',y = 'Text',data = df_dv_ts, label = 'Domestic violence Tweets')
legend = ax1.legend(loc='upper left')
plt.ylabel('Tweet count')
plt.title('Domestic Violence VS Subtance Abuse tweets')
plt.show()
Fbprophet¶
What is Prophet?
- “Prophet” is an open-sourced library available on R or Python which helps users analyze and forecast time-series values released in 2017.
- With developers’ great efforts to make the time-series data analysis be available without expert works, it is highly user-friendly but still highly customizable, even to non-expert users.
The Prophet Forecasting Model We use a decomposable time series model with three main model components: trend, seasonality, and holidays. They are combined in the following equation:
y(t) = g(t) + s(t) + h(t) + εt
- g(t): piecewise linear or logistic growth curve for modelling non-periodic changes in time series
- s(t): periodic changes (e.g. weekly/yearly seasonality)
- h(t): effects of holidays (user provided) with irregular schedules
εt: error term accounts for any unusual changes not accommodated by the model
Using time as a regressor, Prophet is trying to fit several linear and non linear functions of time as components.
- Modeling seasonality as an additive component is the same approach taken by exponential smoothing in Holt-Winters technique .
- We are, in effect, framing the forecasting problem as a curve-fitting exercise rather than looking explicitly at the time based dependence of each observation within a time series.
In [126]:
from fbprophet import Prophet
from fbprophet.plot import plot_plotly
import plotly.offline as py
In [127]:
df1 = sa_df_ts.rename(columns={'Date': 'ds',
'Text': 'y'})
df1.head()
Out[127]:
In [129]:
df2 = df_dv_ts.rename(columns={'date': 'ds',
'Text': 'y'})
df2.head()
Out[129]:
Substance abuse
In [136]:
my_model = Prophet(interval_width=0.95,daily_seasonality = True)
In [137]:
my_model.fit(df1)
Out[137]:
In [138]:
future_dates = my_model.make_future_dataframe(periods=12, freq='MS')
future_dates.head()
Out[138]:
In [139]:
forecast = my_model.predict(future_dates)
forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].head()
Out[139]:
In [140]:
my_model.plot(forecast, uncertainty=True)
Out[140]:
Changepoints
- Another question to answer is whether my time series encounters any underlying changes in the phenomena e.g. a new product launch, unforeseen calamity etc.
- At such points, the growth rate is allowed to change.
- These changepoints are automatically selected.
- However, a user can also feed the changepoints manually if it is required.
- In the below plot, the dotted lines represent the changepoints for the given time series.
Changepoints are the datetime points where the time series have abrupt changes in the trajectory.
By default, Prophet adds 25 changepoints to the initial 80% of the data-set.
In [144]:
plt.style.use('default')
from fbprophet.plot import add_changepoints_to_plot
fig = my_model.plot(forecast)
a = add_changepoints_to_plot(fig.gca(), my_model, forecast)
In [143]:
pro_change= Prophet(n_changepoints=20, yearly_seasonality=True)
forecast = pro_change.fit(df1).predict(future_dates)
fig= pro_change.plot(forecast);
a = add_changepoints_to_plot(fig.gca(), pro_change, forecast)
Domestic violence
In [150]:
my_model = Prophet(interval_width=0.95,daily_seasonality = True)
In [151]:
my_model.fit(df2)
Out[151]:
In [152]:
future_dates = my_model.make_future_dataframe(periods=12, freq='MS')
future_dates.head()
Out[152]:
In [153]:
forecast = my_model.predict(future_dates)
forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].head()
Out[153]:
In [154]:
my_model.plot(forecast, uncertainty=True)
Out[154]:
In [155]:
plt.style.use('default')
from fbprophet.plot import add_changepoints_to_plot
fig = my_model.plot(forecast)
a = add_changepoints_to_plot(fig.gca(), my_model, forecast)
In [156]:
pro_change= Prophet(n_changepoints=20, yearly_seasonality=True)
forecast = pro_change.fit(df1).predict(future_dates)
fig= pro_change.plot(forecast);
a = add_changepoints_to_plot(fig.gca(), pro_change, forecast)
Domestic Violence Organizations¶
Domestic Violence Organizations
We researched all the organizations that help victims of domestic violence and we came across 100 organizations.
In [70]:
df = pd.read_excel('Domestic Violence Associations.xlsx',engine='openpyxl')
df
Out[70]:
In [71]:
import plotly.express as px
import numpy as np
fig = px.sunburst(df, path=['Category', 'Type of Organizations','Organizations'],
color_continuous_scale='RdBu',width=800, height=1000)
#color_continuous_midpoint=np.average(df['lifeExp'], weights=df['pop']))
fig.show()
#fig.write_html("Association.html")
Future Work¶
Future work:
- More Exploratory Data Analysis on the twitter data
- Descriptive Statistics and data visualization
- Try to consider other important factors that cause domestic violence
- Try to scrape data for Domestic violence organizations on twitter.
- BERT topic modeling
- Run some classification algorithms to detect Domestic violence and substance abuse
For Dataset and other information related to the project, click on my name Sagar Mehta and find the link to my github profile, where you will find everything.
In [ ]: